Group  Reports  record  the  findings  and  results  of  IDA  established  working  groups  and 
panels  composed  of  senior  individuals  addressing  major  issues  which  otherwise  would  be 
the  subject  of  an  IDA  Report.  IDA  Group  Reports  are  reviewed  by  the  senior  individuals 
responsible  for  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  and 
relevance  to  the  problems  studied,  and  are  released  by  the  President  of  IDA. 


Papers,  also  authoritative  and  carefully  considered  products  of  IDA,  address  studies  that 
are  narrower  in  scope  than  those  covered  In  Reports.  IDA  Papers  are  reviewed  to  ensure 
that  they  meet  the  high  standards  expected  ol  refereed  papers  In  professional  journals  or 
formal  Agency  reports. 
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ceived  added  expense  and  the  difficulties  associated  with  collecting  and  understanding  the  measures.  Senior  managers 
can  contribute  significantly  to  the  evolution  of  metrics  use  by  supporting  training  for  acquisition  personnel  along  with  con¬ 
tinuing  technical  support  for  them  to  implement  metrics  effectively  in  their  programs. 
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Provide  data  and  analysis  to  strengthen  OSD  leadership 
Improve  Program  Managers’  risk  management  capabilities 
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ware  measurement  for  risk  management  in  acquisition  programs.  This  recommendation  would  directly  affect  the  architects 
of  the  DoD  S&T  Plan. 
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From  recent  experience,  it  is  estimated  to  cost  approximately  $35K  to  $1 00K  to  conduct  an  evaluation  of  one  contractor  s 
process,  depending  on  the  team  source,  size,  and  the  complexity  of  program  requirements. 


2.  EVALUATE  CONTRACTORS’  SOFTWARE 
PROCESSES  FOR  ACQUISITION  DECISIONS 
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WHEN  Estimate  3  months  to  develop  guidance,  then  3  months  to  implement 
COST  Estimate  $35K  to  $1 00K  per  evaluation 


3.  CONDUCT  INDEPENDENT  SOFTWARE  PRODUCT 
RISK  ASSESSMENT  FOR  MILESTONE  2  REVIEW 
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COST  Estimate  up  to  $300K  per  assessment 


4.  ASSESS  TECHNICAL  RISK  REDUCTION  AND  SET 
FUTURE  GOALS  AT  MILESTONE  REVIEWS 
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menting  this  within  DoD  programs  should  be  minimal  since  it  is  an  expected  part  of  good  system  engineering  practice.  It  will 
help  to  reduce  life  cycle  cost  by  identifying  and  resolving  risks  early,  and  improving  the  thoroughness  of  DAB  reviews. 


4.  ASSESS  TECHNICAL  RISK  REDUCTION  AND  SET 
FUTURE  GOALS  AT  MILESTONE  REVIEWS 
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COST  Minimal,  expected  from  DoDI  5000.2  and  good  system  engineering  practice 


5.  REPORT  BASIC  SOFTWARE  METRICS  and  RISKS 
TO  AID  DoD-WIDE  IMPROVEMENT  ANALYSES 
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proximately  six  months  to  draft  the  charter  and  first-year  operating  plan  of  the  improvement  analysis  activity.  It  is  envi¬ 
sioned  that  approximately  four  full-time  staff  years  of  labor  would  suffice  for  initial  analyses.  Once  the  reporting  channel 
(Recommendation  5)  is  working  and  data  becomes  available,  the  amount  of  effort  would  increase  for  more  thorough  analy 
ses. 
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COST  Estimate  initial  effort  about  4  staff  years  per  year,  increasing  as  significant  pro¬ 
gram  data  accrues 


7.  RETAIN  SOFTWARE  EXPERIENCE  DATA 
FOR  LONG-TERM  IMPROVEMENT  ANALYSES 


Slide  28  of  46 


7.  RETAIN  SOFTWARE  EXPERIENCE  DATA  FOR 
LONG-TERM  IMPROVEMENT  ANALYSES 


r 


~o 

'0 

o 

•4— > 

0 

0 

T3 

f  ■ 

O 

0 

O 

CL 

"D 

3 

0 

0 

O 

CO 

> 

a) 

0 

*4— ' 

E 

'c 

3 

0 

4—* 

0 

O 

£ 

■4—* 

CL 

CL 

O 

O 

0 

4— ■» 

3 

O 

>N 

0 

E 

0 

o 

> 

.t; 

O 

0 

l_ 

O 

CL 

CL 

E 

0 

V 

o 

O 

0 

0 

0 

> 

0 

Lc 

>> 

o 

0 

3 

0 

0 

3 

0 

E 

0 

0 

> 

"V 

o 

CD 

> 

3 

LU 

O 

~o 

o 

E 


CO 

0 

‘co 

0 

T3 

03 

c 

03 

0 

O 

3 

0 

c 

0 

0 

o 

0 

O 

0 

‘co 

CL 

X 

_Q 

b 

0 

O 

<5 

cr 

o 

03 

o 

< 

0 

;g 

0 

CD 

o 

£ 

4  * 

> 

i 

a 

0 

O 

0 

CL 

E 

CL 

E 

.3 
■*— < 

O 

o 

o 

o 

E 

o 

3 

0 

O 

4—* 

T3 

0 

0 

0 

T3 

O 

o 

0 

0 

"D 

O 

o 

~o 

0 

0 

3 

0 

i_ 

E 

■4— > 
0 
O 
O 

0 

3 

0 

JD 

T3 

0 

0 

0 

3 

0 

0 

O 

O 

CD 

o 

0 

•*—> 

0 

T“5 

£ 

4—* 

o 

w 

4— » 

O 

3 

Q. 

c 

CD 

0 
4— » 

3 

3 

0 

3 

0 

0 

£ 

3 

O 

0 

■o 

3 

CL 

0 

4— ■ 

T3 

0 

O 

O 

0 

C 

03 

CL 

0 

o 

0 

JD 

_0 

lc 

O 

"0 

3 

E 

0 

‘0 

> 

O 

4— » 

CO 

d) 

0 

CD 

O 

i— . 

< 

0~ 

CL 

cvi 

0 

CO 

<: 

_o 

CD 

JD 

3 

_0 

CL 

3 

o 

~o 

CD 

co 

0 

_Q 


JD 


CD 

3 

o 

CL 

CD 

0 

L_ 

T3 

03 

_0 

0 

"CO 

CD 

_0 

0 

TJ 

T3 

3 

0 

0 

N 

O 


3 

0 

O 


o3 

< 

Q 

CD 

=) 


-  £2 

0  o 

£  3 

0  _Q 
g)  '3 

2  c 
£  8 
E  Q 
i  Q 


T3 

3 

0 


8. « 

Q.  Q 

8  § 

0  03 

0  tC 

c  o 

0  CL 
^  Q_ 
3  3 

0  0 

E  o 

0 

>  > 

o  = 
>-  0 
CL  '4= 

E  c 

« 

0  0 

£  ^ 

> 

£  0 
’>  0 


0 

4— » 

0 

■Q 


0 
3 

!q 

E  f 

o  t 
O  3 

i-  c\i 


0 


3 

o 

E 

CD 


CD 

3 

O 

4— ' 
0 
T3 
3 
0 

E 

E 

o 

o 

0 

DC 


o 

D) 

3 

‘c 

3 

_0 

Q. 


0 

0 

■g 

'o 

3 

‘o 

O 


T3 

3 

0 

CO 

LU 

< 

Q 

CD 

3 

> 


o 

CD 

05 

CO 

3 

£ 

E 

V 

03 

0 

CD 

4—* 

1 

> 

CD 

i— 

3 

CD 

JO 

CL 

03 

o 

CD 

> 

0 

H— 

CD 

1c 

3 

4—' 

CO 

ID 

3 

CD 

c 

o 

oC 

0 

o 

0 

4—* 

•; — 

o 

0 

0 

0 

E 

o 

CD 

"o 

< 

> 

Q 

0) 

lc 

CD 

4— » 

0 

‘c 

0 

a) 

4— » 

0 

03 

T5 

E 

0 

4—* 

CO 

.3 

LU 

O 

D 


LU 

X 

5 


CO 

o 

a 


Slide  29  of  46 


TECHNOLOGY  INTERESTS  FROM  PHASE  I  SURVEY 
AND  SUBSEQUENT  INTERVIEWS 


Slide  30  of  46 


TECHNOLOGY  INTERESTS  FROM  PHASE  I  SURVEY 
AND  SUBSEQUENT  INTERVIEWS 


Metrics  validation 


TECHNICAL  NEEDS  RELATED 
TO  THE  METRICS  PLAN 


.9  -o 

—  c 


CD  -p; 

N  ™ 

•e  .91 

O  _Q 
05  o 

S  -C 

03  O 
O  03 
CD 


CD  C 
-C  CD 

"o  E 

CD  CD 


£Z 

CD  O 


S'g 

CD 

O  ® 

c  E 

03  £ 
CO  2- 


^  to 

CD 

CO  ~0 


>  Q. 
CD  tr 

c  3 


CD  CD 
>  "O 


—  C 

03 

CD  £ 

E  co 


™  c 

CD  03 
Q_ 
O  CO 

.9>:F 


0)  § 

CO  •  — 


Q_  C 
0 

-E  E 

TO  0 
0  Ql 

|  E 

0  CD 


0  T3 


^  O 

-o  O’ 
0  0 
C  *“ 

.9  o 
■*  c 
c  -2 

®  C5 
CD  O 
jr  r»= 
o 

*S  CD 
ro  q_ 

£  co 


E  to 

I  ® 
8  g- 
0 

"  0 


ss  i 

■  ■  «  -5 

c  (C  ^ 

9o  ^ 
~  05  O 
.E  cd  ^ 


0  O 
U  0 


05  > 

_9  0 

O  -C 


™  to 

0  O 


cd  £- 

_c 


O  0 
o  CJ 


o  CD 
O  CL 


lo  cd  co 


Metrics  validation:  Recommendations  1  through  4 


TECHNICAL  NEEDS  RELATED 
TO  THE  METRICS  PLAN 


O 


o  tl 
Q>  O 
Q-5Z 
.>*LU 


(/) 

•a 

o> 

o 


o 
■  ■■■ 

m 

O 

0) 

Q. 

CO 


>* 

D) 

O 


CD 

a> 


o 

0) 


c 

o 

si 

o  £ 
E  E 
££ 


<D  r 

co  O) 
3  0) 

o  ® 

Sg 


CO 
CD 

o  a> 

'B  £ 

to  a> 

E  w 
v.  o 


CO 


CD 

£  ■o 

CD  CD 
C  T3 
!3  C 
a>  cd 

■o  « 

£  >> 

2^ 

O  3 

Si 

i  I 

a  a! 

fc.  ll  cm 

ODCO) 

©2v 

O  3  in 

PI 

3  .E  -D 

a)  a,  j= 

g|5 

O  u  r 

E  £  co 
E  £  CO 

°£iu 
U  ..Hi 
O  LU 


c 

a> 

E 

o 

D) 

0 

c 

CD 

E 

co 

*z 

c 

o 

« 

‘5 

a- 

a 

CD 

c 

0 

CO 

=J 

it. 

0 

£ 

■o 

c 

0 

co 

o 


0 

E 


o 

'co 

■> 

Q 

o 

D 


C:  =  eg 


c 0 
a 

0  o 

e| 

CD  C 
Jr 

O  a) 
O  -D 


c 

_  .2 

S  £ 

13  D 

o  £ 

2  = 
o!£ 


co  2 
o  ® 
•=£ 
0  o 

E  2 

0)  CL 

>  TJ 
c 
o  rc 
0 

5z  w 
0  0 
o 

O  O) 
*•-  ~ 
“O  c 
0  0 

® ! 
lo¬ 
co  o 

0  > 
—  0 

£  T5 
o  3** 

m  0 

0  X 
CL"^ 
0  O 
a  *- 
—  TJ 
O  0 

o  % 
+-*  0 

O  0 

'co 

0  c 
n  o 

c  IS 
o  « 

0  c 
o  0 

=  E 
■§£ 
3  | 

O  .E 


m 

o 

o 

0) 

o 


0 


0 

a 

§2 

a.  0 
x  c 

LU  0 


C/) 

0 

k_ 

3 

CO 

0 

0 

E 

0 

o 

c 

0 

E 

k_ 

o 

t 

0 

a. 

o 

=3 

"D 

O 


0 

Sc 

T3  JJ> 

£  E 
0  a) 

C  O) 

o  ® 

£  C 

o  5 

0  E 
0  j* 

CO  CO 

c 

O  c 
0  o 
a  ~ 

l§£ 

TJ  3 
•=  O’ 
3  O 

O  0 


CO 


0 
a 
c 

2  0 

i  3 

O  CO 

t 

0  £ 
0-  E 


.c 

a 

0 

0 

co 

0 

DC 


c 

0 

E 

0 

O) 

0 

c 

0 

E 

CO 

c 

0 

k. 

0 

| 

o 

CO 

TJ 

0 

CO 

0 

-Q 

■ 

CO 

a 


0 

E 


CO 

0 

b 

3 

CO 

0 

CO 

0 

a 

TJ 

c 

0 

0 

o 

c 

0 

TJ 

3 

O 


CO 

TJ 

0 

c 

o 

'co 

'o 

0 

Q 


0 

0 

CO 

0 

DC 


CO 

0 

'&) 

o 

o 

TJ 

o 

£ 

0 

E 

o 

O 


c 

0 

E 

0 

O) 

0 

c 

0 

E 

CO 


CO 

a 


0 

E 

0 

k— 

o 

a 


c 

o 


0 

Q 


CO 

o 


0 

E 

O 

O 


-C 

a 

i. 

0 

0 

CO 

0 

DC 


E 

D) 

!j 

TO  Q 
Q.— 

II 

£1 

a  o 

<D  Q. 

£i 

,E  o 
°  co 
£>|- 

°8 

O  *5; 

O)  0 
C  CO 
3 

3  i5 
o 

O  CO 
O  TO 
TO  CD 

co  a 
®.E 

T3  „ 
O  CO 

E  "o 
c  o 
c  £ 

£  CD 

TO  E 

.io 

coO 
©  r 

"O  D) 
©  © 
O  CO 

I-  4-» 

Q.ltz 


c 

o 

0 

E 

2i2 

0 

Jl 

CO  ^ 

o  o 
O  E 


c 

o 

si 

4-I  3 

o  E 

0  E 
0l£ 


“O 

c 

0 

°o 

CO 


O) 


0 


LU 


> 

^  CO 

■o 

0  x: 

£  3 
TO  CO 

TO  TO 
S5  T3 

~  o 
5  E 

TO  4- 

a.  c 

TO  TO 

°  E 

w  to 

O  (/) 
'Z  TO 

®  CO 

E  To 

<D  >* 
>  £ 

a  5 
0  0 
3=  O- 
a>  0 
o 

O  o 

S  w 

S£ 

3  2 

O  a. 


co 

o 


0 

E 


o 


0 

o 

C  CO 
0 

■3  w 
0  ^ 
Q-  0 

x  c 

LU  0 


T3 

C 

0 

0 

0 

“O 

o 

91 

o 

D- 

O 

o> 

c 

’> 

!E 

a 

k_ 

0 

T3 

C 

0 

c 

o 

0 

N 

C  CU 

0  £ 
O)  3 

O  0 
^  a 


o  >» 
bn3  =: 

CO  c  -Q 
■-0  0 

cr  0  iS 

0  ^  0 

k.  3  r 
O  —  o 

H—  L.  W 

to  £  w 

Is  8 


0 
CD  E 


co 

XJ 

k. 

o 

a 

0 

>S 

a 

0 

O) 

0 


0 


0 
0 
c n 


c 

o 

0 

0 

> 

CO 

o 

c 

0 


Slide  33  of  46 


8.  PROVIDE  FOCUS  FOR  RESEARCH  SUPPORTING 
DoD-WIDE  METRICS  USE  IN  RISK  MANAGEMENT 


O 


Slide  34  of  46 


8.  PROVIDE  FOCUS  FOR  RESEARCH  SUPPORTING 
DoD-WIDE  METRICS  USE  IN  RISK  MANAGEMENT 


0 

TD 


I 

O 


Q 

o 

O 


c 
0 

E 

0 

2  E 


£ 

CO 

‘k- 

0 

c 

JZ 

0 

CO 

c 

CO 

05 

*13 

CL 

cr 

0 

| — 

03 

o3 

i— 

C/) 

^0 

c 

CO 

•  "" 

0 

0 

’kl 

0 

0) 

0 

0 

E 

i_ 

H”  1 

CL 

O 

T3 

c 

C 

0 

0 

-t— • 
0 

0 

0 

O 

O 

CL 

CD 

X 

0 

T5 

C 

C 

0 

0 

g 

0 

O 

X 

C 

i_ 

0 

0 

0 

CL 

O 

0 

O 

O 

1 — 

0 

OS 

‘o 


O  § 


O)  co 

•If 

£  3 

I  ® 

X3  H— ' 

Z3  ^ 
0  ® 
0 
0 


~o 

C 
0  C 

>>  !c 
o 


0 

O 

O 

> 

T3 

0 

0 

•g 

i 

Q 

o 

O 


<: 

0 

O 

0^ 

o 

L. 

CL 

_o 

Q. 

O 


0  g 
g  c 

O  (D 


o 


0 

c 

0 

0 

E 


0 

O 

X 

0 

O 

0 

0 


c 

0 

o 


0 
TD 

0 

C 

O  0 
O  _0 

cvi  2 


< 

i 


0 

E 

0 

CD 


0  O 

E  a. 

■D  £ 
C 
0 

0 
3 
O 
O 


CD 

JO 

O 

c 

_C 

o 

0 


c 

o 

0  C 

C  g 

®  ® 

I  ® 

CL 

E 


o 

0 


o  g 

0 

■=  c 

D  =5 
*0  >. 

>  *0 

©  C 
tz  0 

1  i 

g  "D 

S-  C 
_C  0 

o  M_ 

V—  -*—* 

0  O 
0  C 

S  .<2 

0  £ 
o  -Q 

If 

C  "D 

c  § 

CD 

t= 

3  O 

O  J5 


> 

X 

5 


0 
O 

c 

0 

g 

3 
CD 

LU 

tr 

Q 
Q 

CD 

c 

O 

=  O 


0 

i_ 

0 

C 

c 

_0 

CL 

I- 

oQ 

C/D 


0 

O 

0 


~o 

c 

0  © 
C  0 
0  C 

o  0 
0 

fe  E 

C/D  © 
C  2 

Z  o 

>  o 

>  o 


j2  0 

O 

0  ^ 


C 

o 

0 

o 


0 

0 


o3 

< 

Q 

CO 

=) 

T3 

C 

0 

CO 

O 

Q 

CO 

< 

c 

0 

L_ 

3 

O 


_Q 

3 

0 

TJ 

0 

•*—> 

C 

0 


CL 

X 

0 


0 

0 

O 

O 

0 


£  Q- 


0 
CL 
3 
O 

L. 

_  CD 

0  CD 
O  c 

D)  *  y> 

f  | 

0  > 

>  0 

0  0 

Q  =D 

c\j 


c 

o 


c 

o 


0  0 

|  I 

8  » 

O  0 

0  g 
•g  > 
g  o 
o  © 
O  0 
.  0 
CO  -Q 


O  5 

i  s 


_0 

O 

o 

CD 

C 

‘c 

c 

_0 

CL 

I — 
oQ 
CO 

X 

0 

c 


0 

O 

c 

0 

■g 

’3 

CD 

0 

E 

0 

1. 

o 

0 

_c 

c 

o 

E 

CO 

0 

0 

E 

0 

LU 


LU 


D 


CD 


LTD 

CO 

a> 

td 

CO 


COST  Minimal  for  planning  effort,  associated  research  costs  not  estimated 
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"The  source  selection  criteria  identified  in  (reference  to  Process  Evaluation  Guide)  divide  software  considerations  into  sev¬ 
eral  sections,  one  of  which  includes  the  results  of  the  software  process  evaluation.  It  identifies  how  the  evaluation  results 
will  be  summarized  in  terms  of  red,  yellow,  green,  and  blue  categories.  The  evaluation  results  will  also  identify  the  offeror's 
strengths  and  weaknesses  in  key  areas  as  listed  in  (reference  to  Process  Evaluation  Guide).” 
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mature.  The  overall  vision  of  the  proposed  plan  is  to  establish  a  department-wide,  bottom-to-top  corporate 
approach  for  collecting  and  using  software  metrics  to  improve  analysis  and  decision-making  in  software 
acquisition  and  risk  management.  Acquisition  reform  and  the  need  for  improved  risk  management  should 
provide  more  specific  goals  to  address  under  this  vision,  comparable  to  industry's  goals  of  marketplace  benefits 
from  using  software  metrics.  The  plan  is  framed  as  eight  specific  recommendations,  including  a  list  of  future 
research  and  development  work  that  would  support  their  implementation. 
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